Techniques for benchmarking pairing strategies in a contact center system

ABSTRACT

Techniques for benchmarking pairing strategies in a contact center system are disclosed. In one embodiment, the techniques may be realized as a method for benchmarking pairing strategies in a contact center system comprising: determining for each contact of a plurality of contacts, an associated plurality of historical contact assignments; determining, for each contact, an associated outcome value; partitioning, for each contact, the associated plurality of historical assignments into a first associated subset assigned using a first pairing strategy and a second associated subset assigned using a second pairing strategy; determining, for each contact, a first portion of the associated outcome value attributable to the first associated subset and a second portion of the associated outcome value attributable to the second associated subset; and outputting a difference in performance between the first and second pairing strategies according to the first and second associated portions of the associated outcome value for each contact.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/413,146, filed May 15, 2019, which is a continuation of U.S. patentapplication Ser. No. 15/691,163, filed Aug. 30, 2017, now U.S. Pat. No.10,298,762, which is a continuation of U.S. patent application Ser. No.15/221,698, filed Jul. 28, 2016, now U.S. Pat. No. 9,774,740, which is acontinuation-in-part of U.S. patent application Ser. No. 15/131,915,filed Apr. 18, 2016, now U.S. Pat. No. 9,712,676, which is acontinuation-in-part of U.S. patent application Ser. No. 12/021,251,filed Jan. 28, 2008, now U.S. Pat. No. 9,712,679, and which is acontinuation-in-part of U.S. patent application Ser. No. 14/727,271,filed Jun. 1, 2015 now U.S. Pat. No. 10,051,125, which is a continuationof U.S. patent application Ser. No. 14/472,998, filed Aug. 29, 2014, nowU.S. Pat. No. 9,215,323, which is a continuation of U.S. patentapplication Ser. No. 12/266,446, filed Nov. 6, 2008, now U.S. Pat. No.8,824,658, each of which is hereby incorporated by reference in itsentirety as if fully set forth herein.

FIELD OF THE DISCLOSURE

This disclosure generally relates to contact centers and, moreparticularly, to techniques for benchmarking pairing strategies in acontact center system.

BACKGROUND OF THE DISCLOSURE

A typical contact center algorithmically assigns contacts arriving atthe contact center to agents available to handle those contacts. Attimes, the contact center may have agents available and waiting forassignment to inbound or outbound contacts (e.g., telephone calls,Internet chat sessions, email) or outbound contacts. At other times, thecontact center may have contacts waiting in one or more queues for anagent to become available for assignment.

In some typical contact centers, contacts are assigned to agents orderedbased on time of arrival, and agents receive contacts ordered based onthe time when those agents became available. This strategy may bereferred to as a “first-in, first-out”, “FIFO”, or “round-robin”strategy.

Some contact centers may use a “performance based routing” or “PBR”approach to ordering the queue of available agents or, occasionally,contacts. PBR ordering strategies attempt to maximize the expectedoutcome of each contact-agent interaction but do so typically withoutregard for utilizing agents in a contact center uniformly.

When a contact center changes from using one type of pairing strategy(e.g., FIFO) to another type of pairing strategy (e.g., PBR), overallcontact center performance will continue to vary over time. It can bedifficult to measure the amount of performance change attributable tousing a new pairing strategy because there may be other factors thataccount for some of the increased or decreased performance over time.

In view of the foregoing, it may be understood that there is a need fora system that enables benchmarking of alternative routing strategies tomeasure changes in performance attributable to the alternative routingstrategies.

SUMMARY OF THE DISCLOSURE

Techniques for benchmarking pairing strategies in a contact centersystem are disclosed. In one particular embodiment, the techniques maybe realized as a method for benchmarking pairing strategies in a contactcenter system comprising: determining, by at least one computerprocessor communicatively coupled to and configured to operate in thecontact center system, for each contact of a plurality of contacts, anassociated plurality of historical contact assignments; determining, bythe at least one computer processor, for each contact, an associatedoutcome value; partitioning, by the at least one computer processor, foreach contact, the associated plurality of historical assignments into afirst associated subset assigned using a first pairing strategy and asecond associated subset assigned using a second pairing strategy;determining, by the at least one computer processor, for each contact, afirst portion of the associated outcome value attributable to the firstassociated subset and a second portion of the associated outcome valueattributable to the second associated subset; and outputting, by the atleast one computer processor, a difference in performance between thefirst and second pairing strategies according to the first and secondassociated portions of the associated outcome value for each contact.

In accordance with other aspects of this particular embodiment, for eachcontact, the associated plurality of historical contact assignments maybe a chronologically-ordered sequence of historical contact assignments.

In accordance with other aspects of this particular embodiment, for eachcontact, relative sizes of the first and second portions may be based onrelative positions of the first and second associated subsets within thechronologically-ordered sequence.

In accordance with other aspects of this particular embodiment, for eachcontact, relative sizes of the first and second portions may be based onrelative sizes of the first and second associated subsets.

In accordance with other aspects of this particular embodiment, thefirst pairing strategy may be a behavioral pairing strategy.

In accordance with other aspects of this particular embodiment, thesecond pairing strategy may be a FIFO or a performance-based routingstrategy.

In accordance with other aspects of this particular embodiment, thecontact center system is configured to cycle among at least the firstand second pairing strategies.

In another particular embodiment, the techniques may be realized as asystem for benchmarking pairing strategies in a contact center systemcomprising at least one computer processor communicatively coupled toand configured to operate in the contact center system, wherein the atleast one computer processor is further configured to perform theabove-described method.

In another particular embodiment, the techniques may be realized as anarticle of manufacture for benchmarking pairing strategies in a contactcenter system comprising: a non-transitory computer processor readablemedium; and instructions stored on the medium; wherein the instructionsare configured to be readable from the medium by at least one computerprocessor communicatively coupled to and configured to operate in thecontact center system and thereby cause the at least one computerprocessor to operate to perform the above-described method.

The present disclosure will now be described in more detail withreference to particular embodiments thereof as shown in the accompanyingdrawings. While the present disclosure is described below with referenceto particular embodiments, it should be understood that the presentdisclosure is not limited thereto. Those of ordinary skill in the arthaving access to the teachings herein will recognize additionalimplementations, modifications, and embodiments, as well as other fieldsof use, which are within the scope of the present disclosure asdescribed herein, and with respect to which the present disclosure maybe of significant utility.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to facilitate a fuller understanding of the present disclosure,reference is now made to the accompanying drawings, in which likeelements are referenced with like numerals. These drawings should not beconstrued as limiting the present disclosure, but are intended to beillustrative only.

FIG. 1A shows a schematic representation of a benchmarking sequenceaccording to embodiments of the present disclosure.

FIG. 1B shows a schematic representation of benchmarking sequenceaccording to embodiments of the present disclosure.

FIG. 2A shows a schematic representation of benchmarking sequenceaccording to embodiments of the present disclosure.

FIG. 2B shows a schematic representation of benchmarking sequenceaccording to embodiments of the present disclosure.

FIG. 3A shows a schematic representation of benchmarking sequenceaccording to embodiments of the present disclosure.

FIG. 3B shows a schematic representation of benchmarking sequenceaccording to embodiments of the present disclosure.

FIG. 3C shows a block diagram of a contact center system according toembodiments of the present disclosure.

FIG. 3D shows a block diagram of a behavioral pairing module accordingto embodiments of the present disclosure.

FIG. 4 shows a block diagram of a contact center according toembodiments of the present disclosure.

FIG. 5 shows a flow diagram of a benchmarking method according toembodiments of the present disclosure.

FIG. 6 depicts a block diagram of a benchmarking module according toembodiments of the present disclosure.

FIG. 7A shows a schematic representation of benchmarking sequenceaccording to embodiments of the present disclosure.

FIG. 7B shows a schematic representation of benchmarking sequenceaccording to embodiments of the present disclosure.

FIG. 8 shows a flow diagram of a benchmarking method according toembodiments of the present disclosure.

DETAILED DESCRIPTION

A typical contact center algorithmically assigns contacts arriving atthe contact center to agents available to handle those contacts. Attimes, the contact center may have agents available and waiting forassignment to inbound or outbound contacts (e.g., telephone calls,Internet chat sessions, email) or outbound contacts. At other times, thecontact center may have contacts waiting in one or more queues for anagent to become available for assignment.

In some typical contact centers, contacts are assigned to agents orderedbased on time of arrival, and agents receive contacts ordered based onthe time when those agents became available. This strategy may bereferred to as a “first-in, first-out”, “FIFO”, or “round-robin”strategy. For example, a longest-available agent pairing strategypreferably selects the available agent who has been available for thelongest time.

Some contact centers may use a “performance based routing” or “PBR”approach to ordering the queue of available agents or, occasionally,contacts. PBR ordering strategies attempt to maximize the expectedoutcome of each contact-agent interaction but do so typically withoutregard for utilizing agents in a contact center uniformly. Some variantsof PBR may include a highest-performing-agent pairing strategy,preferably selecting the available agent with the highest performance,or a highest-performing-agent-for-contact-type pairing strategy,preferably selecting the available agent with the highest performancefor the type of contact being paired.

For yet another example, some contact centers may use a “behavioralpairing” or “BP” strategy, under which contacts and agents may bedeliberately (preferentially) paired in a fashion that enables theassignment of subsequent contact-agent pairs such that when the benefitsof all the assignments under a BP strategy are totaled they may exceedthose of FIFO and PBR strategies. BP is designed to encourage balancedutilization of agents within a skill queue while neverthelesssimultaneously improving overall contact center performance beyond whatFIFO or PBR methods will allow. This is a remarkable achievementinasmuch as BP acts on the same calls and same agents as FIFO or PBRmethods, utilizes agents approximately evenly as FIFO provides, and yetimproves overall contact center performance. BP is described in, e.g.,U.S. patent application Ser. No. 14/871,658, filed Sep. 30, 2015, whichis incorporated by reference herein. Additional information about theseand other features regarding the pairing or matching modules (sometimesalso referred to as “SATMAP”, “routing system”, “routing engine”, etc.)is described in, for example, U.S. Pat. No. 8,879,715, which isincorporated herein by reference.

Some contact centers may use a variety of other possible pairingstrategies. For example, in a longest-available agent pairing strategy,an agent may be selected who has been waiting (idle) the longest timesince the agent's most recent contact interaction (e.g., call) hasended. In a least-occupied agent pairing strategy, an agent may beselected who has the lowest ratio of contact interaction time to waitingor idle time (e.g., time spent on calls versus time spent off calls). Ina fewest-contact-interactions-taken-by-agent pairing strategy, an agentmay be selected who has the fewest total contact interactions or calls.In a randomly-selected-agent pairing strategy, an available agent may beselected at random (e.g., using a pseudorandom number generator). In asequentially-labeled-agent pairing strategy, agents may be labeledsequentially, and the available agent with the next label in sequencemay be selected.

In situations where multiple contacts are waiting in a queue, and anagent becomes available for connection to one of the contacts in thequeue, a variety of pairing strategies may be used. For example, in aFIFO or longest-waiting-contact pairing strategy, the agent may bepreferably paired with the contact that has been waiting in queue thelongest (e.g., the contact at the head of the queue). In arandomly-selected-contact pairing strategy, the agent may be paired witha contact selected at random from among all or a subset of the contactsin the queue. In a priority-based routing or highest-priority-contactpairing strategy, the agent may be paired with a higher-priority contacteven if a lower-priority contact has been waiting in the queue longer.

Contact centers may measure performance based on a variety of metrics.For example, a contact center may measure performance based on one ormore of sales revenue, sales conversion rates, customer retention rates,average handle time, customer satisfaction (based on, e.g., customersurveys), etc. Regardless of what metric or combination of metrics acontact center uses to measure performance, or what pairing strategy(e.g., FIFO, PBR, BP) a contact center uses, performance may vary overtime. For example, year-over-year contact center performance may vary asa company shrinks or grows over time or introduces new products orcontact center campaigns. Month-to-month contact center performance mayvary as a company goes through sales cycles, such as a busy holidayseason selling period, or a heavy period of technical support requestsfollowing a new product or upgrade rollout. Day-to-day contact centerperformance may vary if, for example, customers are more likely to callduring a weekend than on a weekday, or more likely to call on a Mondaythan a Friday. Intraday contact center performance may also vary. Forexample, customers may be more likely to call at when a contact centerfirst opens (e.g., 8:00 AM), or during a lunch break (e.g., 12:00 PM),or in the evening after typical business hours (e.g., 6:00 PM), than atother times during the day. Intra-hour contact center performance mayalso vary. For example, more urgent, high-value contacts may be morelikely to arrive the minute the contact center opens (e.g., 9:00 or9:01) than even a little later (e.g., 9:05). Contact center performancemay also vary depending on the number and caliber of agents working at agiven time. For example, the 9:00-5:00 PM shift of agents may perform,on average, better than the 5:00-9:00 AM shift of agents.

These examples of variability at certain times of day or over largertime periods can make it difficult to attribute changes in performanceover a given time period to a particular pairing strategy. For example,if a contact center used FIFO routing for one year with an averageperformance of 20% sales conversion rate, then switched to PBR in thesecond year with an average performance of 30% sales conversion rate,the apparent change in performance is a 50% improvement. However, thiscontact center may not have a reliable way to know what the averageperformance in the second year would have been had it kept the contactcenter using FIFO routing instead of PBR. In real-world situations, atleast some of the 50% gain in performance in the second year may beattributable to other factors or variables that were not controlled ormeasured. For example, the contact center may have retrained its agentsor hired higher-performing agents, or the company may have introduced animproved product with better reception in the marketplace. Consequently,contact centers may struggle to analyze the internal rate of return orreturn on investment from switching to a different to a differentpairing strategy due to challenges associated with measuring performancegain attributable to the new pairing strategy.

In some embodiments, a contact center may switch (or “cycle”)periodically among at least two different pairing strategies (e.g.,between FIFO and PBR; between PBR and BP; among FIFO, PBR, and BP).Additionally, the outcome of each contact-agent interaction may berecorded along with an identification of which pairing strategy (e.g.,FIFO, PBR, or BP) had been used to assign that particular contact-agentpair. By tracking which interactions produced which results, the contactcenter may measure the performance attributable to a first strategy(e.g., FIFO) and the performance attributable to a second strategy(e.g., PBR). In this way, the relative performance of one strategy maybe benchmarked against the other. The contact center may, over manyperiods of switching between different pairing strategies, more reliablyattribute performance gain to one strategy or the other.

Several benchmarking techniques may achieve precisely measurableperformance gain by reducing noise from confounding variables andeliminating bias in favor of one pairing strategy or another. In someembodiments, benchmarking techniques may be time-based (“epochbenchmarking”). In other embodiments, benchmarking techniques mayinvolve randomization or counting (“inline benchmarking”). In otherembodiments, benchmarking techniques may be a hybrid of epoch and inlinebenchmarking.

In epoch benchmarking, as explained in detail below, the switchingfrequency (or period duration) can affect the accuracy and fairness(e.g., statistical purity) of the benchmark. For example, assume theperiod is two years, switching each year between two differentstrategies. In this case, the contact center may use FIFO in the firstyear at a 20% conversion rate and PBR in the second year at a 30%conversion rate, and measure the gain as 50%. However, this period istoo large to eliminate or otherwise control for expected variability inperformance. Even shorter periods such as two months, switching betweenstrategies each month, may be susceptible to similar effects. Forexample, if FIFO is used in November, and PBR is used December, someperformance improvement in December may be attributable to increasedholiday sales in December rather than the PBR itself.

In some embodiments, to reduce or minimize the effects of performancevariability over time, the period may be much shorter (e.g., less than aday, less than an hour, less than twenty minutes). FIG. 1A shows abenchmarking period of ten units (e.g., ten minutes). In FIG. 1A, thehorizontal axis represents time, and the verticl axis represents whethera first pairing strategy (“1”) or a second pairing strategy (“0”) isused. For the first five minutes (e.g., 9:00-9:05 AM), the first pairingstrategy (e.g., BP) may be used. After five minutes, the contact centermay switch to the second pairing strategy (e.g., FIFO or PBR) for theremaining five minutes of the ten-minute period (9:05-9:10 AM). At 9:10AM, the second period may begin, switching back to the first pairingstrategy (not shown in FIG. 1A). If the period is 30 minutes (i.e., eachunit of time in FIG. 1A is equal to three minutes), the first pairingstrategy may be used for the first 15 minutes, and the second pairingstrategy may be used for the second 15 minutes.

With short, intra-hour periods (10 minutes, 20 minutes, 30 minutes,etc.), the benchmark is less likely to be biased in favor of one pairingstrategy or another based on long-term variability (e.g., year-over-yeargrowth, month-to-month sales cycles). However, other factors ofperformance variability may persist. For example, if the contact centeralways applies the period shown in FIG. 1A when it opens in the morning,the contact center will always use the first strategy (BP) for the firstfive minutes. As explained above, the contacts who arrive at a contactcenter the moment it opens may be of a different type, urgency, value,or distribution of type/urgency/value than the contacts that arrive atother times of the hour or the day. Consequently, the benchmark may bebiased in favor of the pairing strategy used at the beginning of the day(e.g., 9:00 AM) each day.

In some embodiments, to reduce or minimize the effects of performancevariability over even short periods of time, the order in which pairingstrategies are used within each period may change. For example, as shownin FIG. 1B, the order in which pairing strategies are used has beenreversed from the order shown in FIG. 1A. Specifically, the contactcenter may start with the second pairing strategy (e.g., FIFO or PBR)for the first five minutes, then switch to the first pairing strategy(BP) for the following five minutes.

In some embodiments, to help ensure trust and fairness in thebenchmarking system, the benchmarking schedule may be established andpublished or otherwise shared with contact center management ahead orother users of time. In some embodiments, contact center management orother users may be given direct, real-time control over the benchmarkingschedule, such as using a computer program interface to control thecycle duration and the ordering of pairing strategies.

Embodiments of the present disclosure may use any of a variety oftechniques for varying the order in which the pairing strategies areused within each period. For example, the contact center may alternateeach hour (or each day or each month) between starting with the firstordering shown in FIG. 1A and starting with the second ordering shown inFIG. 1B. In other embodiments, each period may randomly select anordering (e.g., approximately 50% of the periods in a given day used theordering shown in FIG. 1A, and approximately 50% of the periods in agiven day use the ordering shown in FIG. 1B, with a uniform and randomdistribution of orderings among the periods).

In the examples of FIGS. 1A and 1B, each pairing strategy is used forthe same amount of time within each period (e.g., five minutes each). Inthese examples, the “duty cycle” is 50%. However, notwithstanding othervariables affecting performance, some pairing strategies are expected toperform better than others. For example, BP is expected to performbetter than FIFO. Consequently, a contact center may wish to use BP fora greater proportion of time than FIFO—so that more pairings are madeusing the higher-performing pairing strategy. Thus, the contact centermay prefer a higher duty cycle (e.g., 60%, 70%, 80%, 90%, etc.)representing more time (or proportion of contacts) paired using thehigher-performing pairing strategy. FIG. 2A shows an example of aten-minute period with an 80% duty cycle. For the first eight minutes(e.g., 9:00-9:08 AM), the first pairing strategy (e.g., BP) may be used.After the first eight minutes, the contact center may switch to thesecond pairing strategy (e.g., FIFO) for the remaining two minutes ofthe period (9:08-9:10) before switching back to the first pairingstrategy again (not shown). If, for another example, a thirty-minuteperiod is used, the first pairing strategy may be used for the firsttwenty-four minutes (e.g., 9:00-9:24 AM), and the second pairingstrategy may be used for the next six minutes (e.g., 9:24-9:30 AM).

As shown in FIG. 2B, the contact center may proceed through sixten-minute periods over the course of an hour. In this example, eachten-minute period has an 80% duty cycle favoring the first pairingstrategy, and the ordering within each period starts with the favoredfirst pairing strategy. Over the hour, the contact center may switchpairing strategies twelve times (e.g., at 9:08, 9:10, 9:18, 9:20, 9:28,9:30, 9:38, 9:40, 9:48, 9:50, 9:58, and 10:00). Within the hour, thefirst pairing strategy was used a total of 80% of the time (48 minutes),and the second pairing strategy was used the other 20% of the time (12minutes). For a thirty-minute period with an 80% duty cycle (not shown),over the hour, the contact center may switch pairing strategies fourtimes (e.g., at 9:24, 9:30, 9:48, and 10:00), and the total remains 48minutes using the first pairing strategy and 12 minute using the secondpairing strategy.

In some embodiments, as in the example of FIG. 1B, the order in whichthe pairing strategies are used within a period may change (not shown),even as the duty cycle (percentage of time within the period that agiven strategy is used) remains the same. Nevertheless, for periodswhich are factors or multiples of 60 minutes (e.g., 10 minutes, 30minutes), periods may always or frequently align to boundaries at thetop of each hour (e.g., new periods begin at 9:00, 10:00, 11:00, etc.),regardless of the ordering of pairing strategies to be used for theperiod at the beginning of a given hour.

In some embodiments, as explained below with references to FIGS. 3A-D,choosing a period such as 11 minutes, 37 minutes, some prime or othernumbers that do not factor into 60-minute intervals, can increase thenumber of periods required before a particular pattern repeats. Instead,the alignment of periods may drift through hours, days, weeks, etc.before repeating. The duration of a cycle through each pairing strategymay align infrequently with to the hours of a day, days of a week, weeksof a month or year, etc.

FIG. 3A shows an example of a single non-factor period of 11 minutes andapproximately a 73% duty cycle, with the first eight minutes using afirst pairing strategy and the last three minutes using a second pairingstrategy. FIG. 3B illustrates six consecutive cycles. For example, atthe top of the first hour on the first day of the week (e.g., Monday at9:00 AM), the first period may begin, aligned on the top of the hour,the first hour of the day, and the first day of the week. The firstperiod may last from 9:00-9:11 AM, followed by the second period from9:11-9:22 AM, and so on, as illustrated in FIG. 3B and Table I below.The sixth period begins at 9:55 and ends at 10:06. The top of the secondhour (10:00 AM), occurs during the sixth cycle and is not aligned withthe beginning of a period. FIG. 3C shows the same six periods as FIG.3B, with the horizontal axis marking time on ten-minute intervals toillustrate the intentional intra-hour misalignment further.

TABLE I Period # Time Period Begins 1 9:00 2 9:11 3 9:22 4 9:33 5 9:44 69:55 7 10:06 

As shown in FIG. 3D and Table II below, the alignment of periods withrespect to the nearest hour continues to drift throughout a day, usingan example of a contact center open from 9:00 AM to 5:00 PM (9:00-17:00hours). The first period of the first hour (9:00 AM) is aligned with thetop of the hour (9:00 AM). The first period of the second hour (10:00AM) begins at 10:06 AM, six minutes after the top of the hour. The firstperiod of the third hour (11:00 AM) begins at 11:01 AM, one minute afterthe top of the hour. It would take 60 periods over 11 hours for thefirst period of an hour to once again align with the top of the hour. Asshown in Table II, a contact center that is open from 9-5 would not bealigned on the hour again until 12:00 PM the following day (1.375eight-hour days later).

TABLE II Hour Time of First Period of Hour 1  9:00 2 10:06 3 11:01 412:07 5 13:02 6 14:08 7 15:03 8 16:09 (next day) 9  9:04 10 10:10 1111:05 12 12:00

Table III below shows the sequence of days and times at which a newperiod begins at the top of the hour. For example, assuming five-dayweeks Monday-Friday with eight-hour days from 9-5, the sequence wouldproceed from aligning on Monday at 9:00 AM, to Tuesday at 12:00 PM, toWednesday at 3:00 PM (15:00), to Friday at 10:00 AM, and so on. As shownin Table III, it would take 2.2 weeks for a contact center that is openfive days per week for eight hours per day to be aligned at thebeginning of a day (e.g., Tuesday at 9:00 AM over two weeks later).

TABLE III Day Next Time Period Starts at Top of Hour Monday  9:00Tuesday 12:00 Wednesday 15:00 Friday 10:00 (next week) Monday 13:00Tuesday 16:00 Thursday 11:00 Friday 14:00 (next week) Tuesday  9:00

Table IV below shows the sequence of days of the week on which a newperiod begins at the top of that day of the week. In this example,assuming five-day weeks Monday-Friday with eight-hour days, the sequencewould proceed from aligning with the beginning of the day on Monday inweek 1, Tuesday in week 3, Wednesday in week 5, and so on. As shown inTable IV, it would take 11 weeks for this contact center to be alignedat the beginning of a Monday again.

TABLE IV Week Next Day Cycle Starts at Top of Day 1 Monday 3 Tuesday 5Wednesday 7 Thursday 9 Friday 12 Monday

Thus, as FIGS. 3A-3D and Tables I-IV have illustrated, selecting anon-factor period for an hour/day/week/etc. boundary may be effectivefor enabling the alignment of periods to “drift” through natural timeboundaries over weeks/months/years. Because the alignment of periodsdrifts, it is less likely for a pattern to arise that confoundsmeasuring relative performance of multiple pairing strategies. In someembodiments, selection of a non-factor period may be combined with othertechniques for reducing the effect of confounding variables onperformance, such as randomizing or otherwise changing the ordering ofpairing strategies within each period or a set of periods.

In some embodiments, the contact center may determine which pairingstrategy to use based on the time at which a pairing request is made fora contact. For example, assume a contact center is benchmarking BP andFIFO using the example of FIG. 1A (ten-minute periods with a 50% dutycycle, starting with BP in the first half and FIFO in the second half).If the contact center requests a pairing at 9:04 AM, the time of thepairing falls in the first half of a period, so the BP strategy may beused. If the contact center requests a pairing at 9:06 AM, the time ofthe pairing falls in the second half of the period, so the FIFO strategymay be used.

In other embodiments, the contact center may determine which pairingstrategy to use based on the time at which a contact arrives. Forexample, assume a contact center is benchmarking BP and FIFO as in thepreceding example. If the first contact arrives at 9:04 AM, the time ofarrival falls in the first half of a period, so the BP strategy may beused for the contact. Even if the first contact must wait in a queue fortwo minutes, and the pairing is not requested until 9:06 AM, the pairingmay still be made using the BP strategy. Moreover, if a second contactarrives at 9:05 AM, while the first contact is still waiting in queue,the second contact may be designated for FIFO pairing. Consequently, at9:06 AM, contact choice under behavioral pairing may be limited to onlythe contacts in queue who arrived during the BP portion of the periodand, in this example, only the first contact to arrive would beavailable.

In embodiments for epoch-based benchmarking in which a contact arriveson a boundary between periods, or on a boundary between switchingpairing strategies within a period, the system may have predeterminedtie-breaking strategies. For example, the boundary may be defined as “ator before” an aforementioned time, or “on or after” an aforementionedtime, etc. For example, if a period is defined to be associated withstrategy “A” from 9:00-9:08 and strategy B from 9:08-9:10, it may meanthat a contact must arrive on or after 9:00 but before 9:08 (e.g.,9:07.99) to be considered within the first part of the period.Alternatively, it may mean that a contact must arrive after 9:00 but ator before 9:08.00 to be considered within the first part of the period.

In some embodiments, inline benchmarking techniques may be used, inwhich pairing strategies may be selected on a contact-by-contact basis.For example, assume that approximately 50% of contacts arriving at acontact center should be paired using a first pairing method (e.g.,FIFO), and the other 50% of contacts should be paired using a secondpairing method (e.g., BP).

In some embodiments, each contact may be randomly designated for pairingusing one method or the other with a 50% probability. In otherembodiments, contacts may be sequentially designated according to aparticular period. For example, the first five (or ten, or twenty, etc.)contacts may be designated for a FIFO strategy, and the next five (orten, or twenty, etc.) may be designated for a BP strategy. Otherpercentages and proportions may also be used, such as 60% (or 80%, etc.)paired with a BP strategy and the other 40% (or 20%, etc.) paired with aFIFO strategy.

From time to time, a contact may return to a contact center (e.g., callback) multiple times. In particular, some contacts may require multiple“touches” (e.g., multiple interactions with one or more contact centeragents) to resolve an issue. In these cases, it may be desirable toensure that a contact is paired using the same pairing strategy eachtime the contact returns to the contact center. If the same pairingstrategy is used for each touch, then the benchmarking technique willensure that this single pairing strategy is associated with the finaloutcome (e.g., resolution) of the multiple contact-agent interactions.In other situations, it may be desirable to switch pairing strategieseach time a contact returns to the contact center, so that each pairingstrategy may have an equal chance to be used during the pairing thatresolves the contact's needs and produces the final outcome. In yetother situations, it may be desirable to select pairing strategieswithout regard to whether a contact has contacted the contact centerabout the same issue multiple times.

In some embodiments, the determination of whether a repeat contactshould be designated for the same (or different) pairing strategy maydepend on other factors. For example, there may be a time limit, suchthat the contact must return to the contact center within a specifiedtime period for prior pairing strategies to be considered (e.g., withinan hour, within a day, within a week). In other embodiments, the pairingstrategy used in the first interaction may be considered regardless ofhow much time has passed since the first interaction.

For another example, repeat contact may be limited to specific skillqueues or customer needs. Consider a contact who called a contact centerand requested to speak to a customer service agent regarding thecontact's bill. The contact hangs up and then calls back a few minuteslater and requests to speak to a technical support agent regarding thecontact's technical difficulties. In this case, the second call may beconsidered a new issue rather than a second “touch” regarding thebilling issue. In this second call, it may be determined that thepairing strategy used in the first call is irrelevant to the secondcall. In other embodiments, the pairing strategy used in the first callmay be considered regardless of why the contact has returned to thecontact center.

One approach to considering prior pairing for inline benchmarkingtechniques is depicted in FIG. 4. FIG. 4 shows a flow diagram ofbenchmarking method 400 according to embodiments of the presentdisclosure. Benchmarking method 400 may begin at block 410.

At block 410, an identifier of a contact (e.g., caller) may beidentified or otherwise determined. In this example, a caller's “BillingTelephone Number” or “BTN” may be identified. This example assumes thata caller uses the same BTN for each call. In other embodiments, otheridentifiers of the contact (e.g., a customer identification number,Internet Protocol (IP) address) may be used instead. Having identifiedthe caller's BTN (or other contact identifier), benchmarking method 400may proceed to block 420.

At block 420, a pseudorandom number generator (PRNG) may be seeded withthe BTN (or other contact identifier). Having seeded the PRNG with theBTN, benchmarking method 400 may proceed to block 430.

At block 430, a pseudorandom number may be generated for the contactusing the seeded PRNG. Because the seed will be the same for a givencontact each time the contact returns to the contact center, thegenerated pseudorandom number will also be the same each time for thegiven contact. Having generated the pseudorandom number, benchmarkingmethod 400 may proceed to block 440.

At block 440, a pairing strategy (e.g., BP or FIFO) may be selected forthe given contact based on the generated pseudorandom number. Forexample, if 50% of contacts should be paired using BP, and the other 50%should be paired using FIFO, the PRNG may be configured to generateeither a 1 or a 0. If the generated pseudorandom number is a 1, thecontact may be designated for BP pairing. If the generated pseudorandomnumber is 0, the contact may be designated for FIFO pairing.

In this way, the contact will always be paired using the same strategyeach time the contact returns to the contact center. The PRNG will beseeded with the same seed (e.g., the contact's BTN) each time, so thePRNG will generate the same pseudorandom number for the contact eachtime. Thus, benchmarking method 400 may select the same pairing strategyfor the contact each time. In this way, it is possible to account forprior pairings without relying on a database or other storage means todetermine whether or how a contact has been previously paired. In thisway, benchmarking method 400 is stateless with respect to whether or howa contact has been previously paired. Having selected a pairing strategyfor the contact, benchmarking method 400 may proceed to block 450.

At block 450, the contact may be paired to an available agent using theselected pairing strategy. When a contact has been paired with anavailable agent, components of the contact center system (e.g.,switches, routers) may connect the contact to the agent. Following (orduring) the contact-agent interaction, the agent may create a record ofthe outcome of the interaction. For example, in a sales queue, the agentmay create an order for the contact. In a technical support queue, theagent may create or modify a service ticket. The contact center systemmay also record information about the interaction, such as the time andduration of a call, the BTN or other identifier of the contact, theagent identifier, and other data. At this point, benchmarking method mayproceed to block 460.

At block 460, an identifier of the selected pairing strategy may beassociated with the record of the contact-agent interaction created atblock 450. In some embodiments, this may happen simultaneously with thecreation of the record. For example, when the contact center systemrecords the time and duration of a call, it may also record whether thecall had been paired using a BP or FIFO pairing strategy. In otherembodiments, another module may create a separate record of the pairing.This module may record the time of the pairing, the contact and agentidentifiers, the pairing strategy used (e.g., BP or FIFO), and any otherdata that may be helpful for later matching the pairing record with therecord of the caller-agent interaction outcome. At some later time, thepairing records may be matched with the caller-agent interaction recordsso that the pairing strategy information may be associated with theoutcome in one record or the other (or both). Following block 460,benchmarking method 400 may end. In some embodiments, benchmarkingmethod 400 may return to block 410, waiting for another contact toarrive.

Another approach to considering prior pairing in combination with epochbenchmarking techniques is depicted in FIG. 5. This type of techniquemay be considered “hybrid inline-epoch benchmarking.” FIG. 5 shows aflow diagram of benchmarking method 500 according to embodiments of thepresent disclosure. Benchmarking method 500 may begin at block 510.

At block 510, a contact (e.g., “contact n”) arrives at the contactcenter at a particular time t. Benchmarking method 500 may proceed toblock 520.

At block 520, it may be determined whether the contact has beenpreviously paired; i.e., whether this contact is returning to thecontact center for a subsequent touch or interaction. This decision maybe made using a variety of techniques. For example, the benchmarkingsystem may look up the contact's records using a contact identifier(e.g., BTN or customer ID) in a database to determine whether and whenthe contact had previously contacted the contact center. Using asuitable technique, the benchmarking system may determine that thecontact had been previously paired and, in some embodiments, whether andhow the prior pairing should influence the current pairing.

In some embodiments, the benchmarking system may preferably pair acontact using the same pairing strategy every time the contact returnsto the contact center. Thus, if contact n was previously paired usingpairing strategy “A” (e.g., BP), benchmarking method 500 may proceed toblock 560 for subsequent pairing using pairing strategy A again.Similarly, if contact n was previously paired using pairing strategy“B”) (e.g., FIFO), benchmarking method 500 may proceed to block 570 forsubsequent pairing using pairing strategy B again.

However, if it is determined at block 520 that contact n has not beenpreviously paired (or, in some embodiments, any prior pairing should notinfluence the current pairing), benchmarking method 500 may proceed tousing epoch benchmarking at block 550.

At block 550, time may be used to determine which pairing strategy touse for contact n. In this example, arrival time t may be used. Ifcontact n arrived during a time period when the benchmarking system ispairing using strategy A, benchmarking method 500 may proceed to block560 for subsequent pairing using strategy A. Similarly, if contact narrived during a time period when the benchmarking system is pairingusing strategy B, benchmarking method 500 may proceed to block 570 forsubsequent pairing using strategy B.

At blocks 560 and 570, contacts may be paired to available agents usingpairing strategies A or B, respectively. In some embodiments, more thantwo pairing strategies may be used (e.g., prior pairings using A, B, C,etc. or epoch benchmarking within time periods using A, B, C, etc.).Once paired, the contact may be routed or otherwise connected to theavailable agent within the contact center system. As described abovewith respect to benchmarking method 400 (FIG. 4), the agent may create arecord of the contact-agent interaction, and the contact center systemmay also create or modify this record. Benchmarking method may proceedto block 580.

At block 580, an identifier to the selected pairing strategy (e.g., A orB) may be associated with the record created at block 560 or 570. Asdescribed above with respect to benchmarking method 400, thisassociation may occur simultaneously with the creation of thecontact-agent interaction record, or it may be matched at a later timewith other records created by a benchmarking module or other module.Following block 580, benchmarking method 500 may end. In someembodiments, benchmarking method 500 may return to block 510, waitingfor another contact to arrive.

By associating the pairing strategy with the outcome as in, for example,benchmarking methods 400 and 500, the outcomes associated with eachpairing strategy may be measured (e.g., averaged, accumulated), and therelative performance of each pairing strategy may be measured (e.g., therelative overall performance gain attributable to pairing using BPinstead of pairing using FIFO). This benchmarking data may be used for avariety of purposes. For example, the data may be used to assess thestrength of one pairing module over another. For another example, thedata may be used to improve the strength of a BP module by providing “BPon” and “BP off” (e.g., FIFO) contact-agent interaction records toenhance the artificial intelligence in the system. For another example,the data may be used for billing. Because the value added by one pairingstrategy over another may be measured accurately and fairly, thisbenchmarking data may be used in a pay-for-performance business model,in which a client pays a pairing strategy vendor a percentage of theactual measured value added by using the vendor's pairing strategy(e.g., when BP is on as opposed to when BP is off).

Specifically, in some embodiments, associated outcome data may be usedto determine an economic value or gain associated with using one pairingstrategy instead of another. In some embodiments, the economic value orgain may be used to determine compensation for a vendor or other serviceprovider providing a module or modules for the higher-performing pairingstrategy creating the economic value. For example, if a contact centerbenchmarks BP against FIFO and determines that, for a given time period(e.g., a day, a week, a month, etc.), that BP performed 5% better thanFIFO on average over the time period, the BP vendor may receivecompensation corresponding to the 5% value added by BP (e.g., apercentage of the 5% additional sales revenue, or a percentage of the 5%additional cost savings, etc.). Under such a business model, a contactcenter owner may forgo capital expenditure or vendor fees, only paying avendor for periods of time in which the vendor demonstrates value addedto the contact center's performance.

FIG. 6 shows a block diagram of a contact center system 600 according toembodiments of the present disclosure. The description herein describesnetwork elements, computers, and/or components of a system and methodfor simulating contact center systems that may include one or moremodules. As used herein, the term “module” may be understood to refer tocomputing software, firmware, hardware, and/or various combinationsthereof. Modules, however, are not to be interpreted as software whichis not implemented on hardware, firmware, or recorded on a processorreadable recordable storage medium (i.e., modules are not software perse). It is noted that the modules are exemplary. The modules may becombined, integrated, separated, and/or duplicated to support variousapplications. Also, a function described herein as being performed at aparticular module may be performed at one or more other modules and/orby one or more other devices instead of or in addition to the functionperformed at the particular module. Further, the modules may beimplemented across multiple devices and/or other components local orremote to one another. Additionally, the modules may be moved from onedevice and added to another device, and/or may be included in bothdevices.

As shown in FIG. 6, the contact center system 600 may include a centralswitch 610. The central switch 610 may receive incoming contacts (e.g.,callers) or support outbound connections to contacts via atelecommunications network (not shown). The central switch 610 mayinclude contact routing hardware and software for helping to routecontacts among one or more contact centers, or to one or more PBX/ACDsor other queuing or switching components within a contact center.

The central switch 610 may not be necessary if there is only one contactcenter, or if there is only one PBX/ACD routing component, in thecontact center system 600. If more than one contact center is part ofthe contact center system 600, each contact center may include at leastone contact center switch (e.g., contact center switches 620A and 620B).The contact center switches 620A and 620B may be communicatively coupledto the central switch 610.

Each contact center switch for each contact center may becommunicatively coupled to a plurality (or “pool”) of agents. Eachcontact center switch may support a certain number of agents (or“seats”) to be logged in at one time. At any given time, a logged-inagent may be available and waiting to be connected to a contact, or thelogged-in agent may be unavailable for any of a number of reasons, suchas being connected to another contact, performing certain post-callfunctions such as logging information about the call, or taking a break.

In the example of FIG. 6, the central switch 610 routes contacts to oneof two contact centers via contact center switch 620A and contact centerswitch 620B, respectively. Each of the contact center switches 620A and620B are shown with two agents each. Agents 630A and 630B may be loggedinto contact center switch 620A, and agents 630C and 630D may be loggedinto contact center switch 620B.

The contact center system 600 may also be communicatively coupled to anintegrated service from, for example, a third party vendor. In theexample of FIG. 6, benchmarking module 640 may be communicativelycoupled to one or more switches in the switch system of the contactcenter system 600, such as central switch 610, contact center switch620A, or contact center switch 620B. In some embodiments, switches ofthe contact center system 600 may be communicatively coupled to multiplebenchmarking modules. In some embodiments, benchmarking module 640 maybe embedded within a component of a contact center system (e.g.,embedded in or otherwise integrated with a switch). The benchmarkingmodule 640 may receive information from a switch (e.g., contact centerswitch 620A) about agents logged into the switch (e.g., agents 630A and630B) and about incoming contacts via another switch (e.g., centralswitch 610) or, in some embodiments, from a network (e.g., the Internetor a telecommunications network) (not shown).

A contact center may include multiple pairing modules (e.g., a BP moduleand a FIFO module) (not shown), and one or more pairing modules may beprovided by one or more different vendors. In some embodiments, one ormore pairing modules may be components of benchmarking module 640 or oneor more switches such as central switch 610 or contact center switches620A and 620B. In some embodiments, a benchmarking module may determinewhich pairing module may handle pairing for a particular contact. Forexample, the benchmarking module may alternate between enabling pairingvia the BP module and enabling pairing with the FIFO module. In otherembodiments, one pairing module (e.g., the BP module) may be configuredto emulate other pairing strategies. For example, a benchmarking module,or a benchmarking component integrated with BP components in the BPmodule, may determine whether the BP module may use BP pairing oremulated FIFO pairing for a particular contact. In this case, “BP on”may refer to times when the BP module is applying the BP pairingstrategy, and “BP off” may refer to other times when the BP module isapplying a different pairing strategy (e.g., FIFO).

In some embodiments, regardless of whether pairing strategies arehandled by separate modules, or if some pairing strategies are emulatedwithin a single pairing module, the single pairing module may beconfigured to monitor and store information about pairings made underany or all pairing strategies. For example, a BP module may observe andrecord data about FIFO pairings made by a FIFO module, or the BP modulemay observe and record data about emulated FIFO pairings made by a BPmodule operating in FIFO emulation mode.

Embodiments of the present disclosure are not limited to benchmarkingonly two pairing strategies. Instead, benchmarking may be performed fortwo or more pairing strategies. FIGS. 7A and 7B depict examples ofbenchmarking systems for three pairing strategies (e.g., benchmarkingFIFO, PBR, and BP).

FIG. 7A shows a schematic representation of benchmarking sequenceaccording to embodiments of the present disclosure. In this epochbenchmarking example, a period is 15 units of time, and each pairingstrategy is used for one-third of the time (5 units). FIG. 7A shows twocomplete periods, cycling among pairing strategies “2”, “1”, and “0”twice over 30 units of time. For example, from 9:00-9:10 AM, FIFO may beused; from 9:10-9:20 AM, PBR may be used; and from 9:20-9:30 AM, BP maybe used. This pattern of FIFO-PBR-BP repeats in the second period.

FIG. 7B shows a schematic representation of benchmarking sequenceaccording to embodiments of the present disclosure. In this epochbenchmarking example, a complete period is 30 time units. A preferredpairing strategy “2” (e.g., BP) is used two-thirds of the time, andother pairing strategies “1” and “0” (e.g., FIFO and PBR) are usedone-sixth of the time each. In this example, each time strategy “2”turns off, pairing strategies “1” and “0” alternately turn on. Forexample, the pattern may be BP-FIFO-BP-PBR. In addition to the examplesof FIGS. 7A and 7B, many other patterns for switching among multiplepairing strategies are possible.

In some embodiments, contact center management or other users may prefera “stabilization period” or other neutral zone. For example, consider acontact center benchmarking BP and FIFO pairing strategies. When thesystem transitions from BP to FIFO (or vice versa), contact centermanagement may be concerned that the effects of one pairing strategy maysomehow influence the performance of another pairing strategy. Toalleviate these concerns about fairness, a stabilization period may beadded.

One technique for implementing a stabilization period may be to excludecontact-agent interaction outcomes for the first portion of contactsafter switching pairing strategies. For example, assume a contact centeris benchmarking BP and FIFO with a 50% duty cycle over 30-minuteperiods. In the aforementioned embodiments (e.g., FIGS. 1A and 1B), BPwould be on for 15 minutes, followed by FIFO for 15 minutes, and all ofthe contact-agent interactions in the 30-minute period would be includedin the benchmarking measurement. With a stabilization period, BP wouldbe on for, e.g., 10 minutes. After 10 minutes, the system would switchto FIFO. However, the first, e.g., 10 minutes would be considered astabilization period, and FIFO pairings made during this period would beexcluded from the benchmark. The last 10 minutes of the period wouldcontinue pairing using FIFO, and these FIFO pairings would be includedin the benchmark.

This pattern is illustrated in FIG. 7A. In this example, instead ofdepicting switching among three pairing strategies “2”, “1”, and “0”,the “1” may represent the stabilization period. Pairing strategy “2”(e.g., BP) may be on for the first five time units. After five timeunits, BP may be switched off, and the other pairing strategy (e.g.,FIFO) may be used for the remaining ten time units. The next five units(“1”) may be excluded as being part of the stabilization period, and thefive time units after that (“0”) may be included as being part of theFIFO benchmarking period.

In some embodiments, the stabilization period may be longer or shorter.In some embodiments, a stabilization period may be used in a FIFO-to-BPtransition instead of, or in addition to, a BP-to-FIFO transition (orany transition between two different pairing strategies).

As noted above, some contacts may require multiple “touches” (e.g.,multiple interactions with one or more contact center agents) to resolvethe contact's needs. For example, an individual may call or otherwisecontact a mortgage bank several times. The first call may be merelyintroductory or informational in nature, the second call may be toevaluate different loan offers, and the third call may be to close(accept) or decline a loan offer. Similarly, some technical support andcustomer service requests may require more than one contact interactionto resolve.

In some situations, it may be desirable to select pairing strategieswithout regard to whether a contact has contacted the contact centerabout the same issue multiple times. Instead, the benchmarking techniquemay be configured to account for each pairing strategy. For example,each pairing strategy may be credited in a manner that fairly accountsfor the extent to which it may have contributed to the series of contactinteractions that resulted in a final outcome or resolution of thecontact's needs.

FIG. 8 shows a flow diagram of benchmarking method 800 according toembodiments of the present disclosure. Benchmarking method 800 may beginat block 810.

At block 810, a contact (e.g., “contact n”) arrives at the contactcenter at a particular time t and a particular iteration i. For example,if this time is the first time contact n has contacted the contactcenter about this particular need, it will be designated the firstiteration (i.e., i=1), or this time may be the second (i=2), third(i=3), . . . i^(th) time calling. Benchmarking method 800 may proceed toblock 820.

At block 820, time may be used to determine which pairing strategy touse for contact n. In these embodiments, even if this is a subsequentarrival such that i>1, the pairing strategy selected for any priorpairing need not influence the current pairing. In this example, arrivaltime t may be used. If contact n arrived during a time period when thebenchmarking system is pairing using strategy A, benchmarking method 800may proceed to block 830 for subsequent pairing using strategy A.Similarly, if contact n arrived during a time period when thebenchmarking system is pairing using strategy B, benchmarking method 800may proceed to block 840 for subsequent pairing using strategy B. Insome embodiments, pairing strategy A may be a behavioral pairingstrategy (behavioral pairing “ON”), and pairing strategy B may be adifferent pairing strategy such as FIFO or performance-based routing(behavioral pairing “OFF”).

At blocks 830 and 840, contacts may be paired to available agents usingpairing strategies A or B, respectively. In some embodiments, more thantwo pairing strategies may be used. Once paired, the contact may berouted or otherwise connected to the available agent within the contactcenter system. As described above with respect to benchmarking methods400 (FIGS. 4) and 500 (FIG. 5), the agent may create a record of thecontact-agent interaction, and the contact center system may also createor modify this record. Benchmarking method may proceed to block 850.

At block 850, an identifier to the selected pairing strategy (e.g., A orB) may be associated with an identifier of the current iteration i forthe contact n within the record (or set of records) created at block 830or 840. As described above with respect to benchmarking methods 400 and500, this association may occur simultaneously with the creation of thecontact-agent interaction record, or it may be matched at a later timewith other records created by a benchmarking module or other module.

In some embodiments, benchmarking method 800 may proceed to block 860.At block 860, it may be determined whether the needs of contact n havebeen resolved. If the needs of contact n have not yet been resolved, orin other embodiments where the outcome of a given call may be matched ata later time, benchmarking method 800 may end or return to block 810 forthe next contact to arrive. If the needs of contact n have beenresolved, benchmarking method 800 may proceed to block 870.

At block 870, the final outcome (e.g., a mortgage loan was closed ordeclined, a sale was completed or canceled) may be associated with therecord or records for each of the i iterations for contact n. Table Vshows an example set of records for four contacts W, X, Y, and Z, whoeach contacted the contact center system three times before reaching afinal resolution. In real-world scenarios, there may be many morecontacts, and the number of contact interactions needed to resolve anindividual contact's needs may vary from one contact to the next,ranging from just one contact interaction to three or even more.

TABLE V Iteration i Contact W Contact X Contact Y Contact Z 1 A B A B 2B B A A 3 A A B B Outcome Sale Sale Sale No Sale

As shown in Table V, and as described above with respect to block 850,the identifiers for the contact (W, X, Y, or Z) and the iteration (1, 2,or 3) are associated with the selected pairing strategy used for thegiven contact interaction (A or B). For contact W, iteration 1 waspaired with strategy A, iteration 2 was paired with strategy B, anditeration 3 was paired with strategy A. For contact X, iteration 1 waspaired with strategy B, iteration 2 was paired with strategy B, anditeration 3 was paired with strategy A. For contact Y, iteration 1 waspaired with strategy A, iteration 2 was paired with strategy A, anditeration 3 was paired with strategy B. For contact Z, iteration 1 waspaired with strategy B, iteration 2 was paired with strategy A, anditeration 3 was paired with strategy B.

Also shown in Table V, and as described above with respect to block 870,the final outcome for the contact may be associated with the records foreach iteration. Contacts W, X, and Y completed sales. Contact Z did notcomplete a sale. In some embodiments, a “no sale” determination may bemade if the contact explicitly states that it does not intend tocomplete a sale. In other embodiments, the contact center system maymake a “no sale” determination after a predetermined number ofiterations have occurred without completing a sale, or after apredetermined amount of time has passed since the most recent iterationwithout completing a sale. In some situations, such as when a contacthas resolved a technical support or customer service need, the finalresolution may be a customer satisfaction survey result or score.

Following block 870, benchmarking method 800 may end. In someembodiments, benchmarking method 800 may return to block 810, waitingfor another contact to arrive (e.g., a different contact, or contact nwith a new need).

On a continuous, hourly, daily, weekly, etc. basis, a difference inperformance among pairing strategies may be determined. In someembodiments, the performance difference may be stratified for eachiteration (the performance difference for all contact interactions ofiteration=1; all contact interactions of iteration i=2; etc.). Althougha final resolution may not have been reached after the first contactinteraction (when i=1), the benchmark for the first stratum may bemeasured using the final outcome, which was previously associated withthat first contact interaction either at block 870 or matched at a latertime with other contact center system records.

Using the example shown in Table V, for the benchmark for the firststratum (i=1), two contacts (W and Y) were paired using strategy A forthe first iteration, and both had final outcomes resulting in a sale,achieving a conversion rate of 100%. The other two contacts (X and Z)were paired using strategy B for the first iteration, and only one (X)had a final outcome resulting in a sale, achieving a conversation rateof 50%.

Similarly, for the benchmark for the second stratum (i=2), two contacts(Y and Z) were paired using strategy A for the second iteration, andonly one (Y) had a final outcome resulting in a sale, achieving aconversation rate of 50%. The other two contacts (W and X) were pairedusing strategy B for the second iteration, and both had final outcomesresulting in a sale, achieving a conversion rate of 100%.

Finally, for the benchmark for the third stratum (i=3), two contacts (Wand X) were paired using strategy A for the third iteration, and bothhad final outcomes resulting in a sale, achieving a conversion rate of100%. The other two contacts (Y and Z) were paired using strategy B forthe third iteration, and only one (Y) had a final outcome resulting in asale, achieving a conversion rate of 50%.

These strata and conversion rates are shown in Table VI below:

TABLE V Iteration/Stratum i Strategy A Strategy B 1 100%  50% 2  50%100% 3 100%  50%

After determining performances or performance differences between eachpairing strategy for each iteration (based on the final outcomes), theperformance differences may be combined. In some embodiments, theperformance differences may be averaged. In the example of Table V, theaverage conversion rate for strategy A is approximately 83%, and theaverage conversion rate for strategy B is approximately 67%. Strategy Aperformed almost 24% better than strategy B. In some embodiments, theperformance differences may be normalized based on the total number ofcontact interactions for each strategy for each iteration.

In some embodiments, some iterations may be weighted more or less thanother iterations. For example, it may be determined (by, e.g., contactcenter system administrators or other business leaders) that the firstcontact interaction is the most critical to determining whether or towhat extent a desirable final outcome will be achieved. In othersituations, it may be determined that the final contact interaction isthe most critical. In these situations, the more critical strata may beweighted more heavily than less critical strata in the finalbenchmarking result.

At this point it should be noted that behavioral pairing in a contactcenter system in accordance with the present disclosure as describedabove may involve the processing of input data and the generation ofoutput data to some extent. This input data processing and output datageneration may be implemented in hardware or software. For example,specific electronic components may be employed in a behavioral pairingmodule or similar or related circuitry for implementing the functionsassociated with behavioral pairing in a contact center system inaccordance with the present disclosure as described above.Alternatively, one or more processors operating in accordance withinstructions may implement the functions associated with behavioralpairing in a contact center system in accordance with the presentdisclosure as described above. If such is the case, it is within thescope of the present disclosure that such instructions may be stored onone or more non-transitory processor readable storage media (e.g., amagnetic disk or other storage medium), or transmitted to one or moreprocessors via one or more signals embodied in one or more carrierwaves.

The present disclosure is not to be limited in scope by the specificembodiments described herein. Indeed, other various embodiments of andmodifications to the present disclosure, in addition to those describedherein, will be apparent to those of ordinary skill in the art from theforegoing description and accompanying drawings. Thus, such otherembodiments and modifications are intended to fall within the scope ofthe present disclosure. Further, although the present disclosure hasbeen described herein in the context of at least one particularimplementation in at least one particular environment for at least oneparticular purpose, those of ordinary skill in the art will recognizethat its usefulness is not limited thereto and that the presentdisclosure may be beneficially implemented in any number of environmentsfor any number of purposes. Accordingly, the claims set forth belowshould be construed in view of the full breadth and spirit of thepresent disclosure as described herein.

1. A method comprising: during a first period of time, associating afirst pairing strategy to a first plurality of contacts; during a secondperiod of time later than the first period of time, associating a secondpairing strategy that is different from the first pairing strategy to asecond plurality of contacts; during a third period of time later thanthe second period of time, associating a third pairing strategy that isdifferent from the second pairing strategy to a third plurality ofcontacts; during a fourth period of time later than the third period oftime, associating a fourth pairing strategy that is different from thefirst and third pairing strategies to a fourth plurality of contacts;determining a first performance measurement based on outcomes of thefirst and third pluralities of contacts; determining a secondperformance measurement based on outcomes of the second and fourthpluralities of contacts; and outputting data that enables a comparisonof the first and second performance measurements, wherein the first,second, third, and fourth periods of time occur within a time intervalof less than 24 hours, wherein the first period of time is determinedprior to the first period of time, wherein the second period of time isdetermined prior to the second period of time, wherein the third periodof time is determined prior to the third period of time, wherein thefourth period of time is determined prior to the fourth period of time,wherein each of a plurality of agents is paired with at least onecontact from the first plurality of contacts, at least one contact fromthe second plurality of contacts, at least one contact from the thirdplurality of contacts, and at least one contact from the fourthplurality of contacts.
 2. The method of claim 1, wherein the timeinterval is less than or equal to 12 hours.
 3. The method of claim 1,wherein the time interval is less than or equal to 6 hours.
 4. Themethod of claim 1, wherein the time interval is less than or equal to 3hours.
 5. The method of claim 1, wherein the time interval is less thanor equal to 1 hour.
 6. The method of claim 1, wherein the plurality ofagents comprises at least 10 agents.
 7. The method of claim 1, whereinthe plurality of agents comprises at least 30 agents.
 8. The method ofclaim 1, wherein the plurality of agents comprises at least 100 agents.9. The method of claim 1, wherein the time interval is selected toreduce bias attributable to effects related to time.
 10. The method ofclaim 1, wherein a combination of the first and third pluralities ofcontacts share a common distribution of contact types with a combinationof the second and fourth pluralities of contacts.
 11. A systemcomprising: at least one computer processor communicatively coupled toand configured to operate in a contact center system, wherein the atleast one computer processor is further configured to: during a firstperiod of time, associate a first pairing strategy to a first pluralityof contacts; during a second period of time later than the first periodof time, associate a second pairing strategy that is different from thefirst pairing strategy to a second plurality of contacts; during a thirdperiod of time later than the second period of time, associate a thirdpairing strategy that is different from the second pairing strategy to athird plurality of contacts; during a fourth period of time later thanthe third period of time, associate a fourth pairing strategy that isdifferent from the first and third pairing strategies to a fourthplurality of contacts; determine a first performance measurement basedon outcomes of the first and third pluralities of contacts; determine asecond performance measurement based on outcomes of the second andfourth pluralities of contacts; and output data that enables acomparison of the first and second performance measurements, wherein thefirst, second, third, and fourth periods of time occur within a timeinterval of less than 24 hours, wherein the first period of time isdetermined prior to the first period of time, wherein the second periodof time is determined prior to the second period of time, wherein thethird period of time is determined prior to the third period of time,wherein the fourth period of time is determined prior to the fourthperiod of time, wherein each of a plurality of agents is paired with atleast one contact from the first plurality of contacts, at least onecontact from the second plurality of contacts, at least one contact fromthe third plurality of contacts, and at least one contact from thefourth plurality of contacts.
 12. The system of claim 11, wherein thetime interval is less than or equal to 12 hours.
 13. The system of claim11, wherein the time interval is less than or equal to 6 hours.
 14. Thesystem of claim 11, wherein the time interval is less than or equal to 3hours.
 15. The system of claim 11, wherein the time interval is lessthan or equal to 1 hour.
 16. The system of claim 11, wherein theplurality of agents comprises at least 10 agents.
 17. The system ofclaim 11, wherein the plurality of agents comprises at least 30 agents.18. The system of claim 11, wherein the plurality of agents comprises atleast 100 agents.
 19. The system of claim 11, wherein the time intervalis selected to reduce bias attributable to effects related to time. 20.The system of claim 11, wherein a combination of the first and thirdpluralities of contacts share a common distribution of contact typeswith a combination of the second and fourth pluralities of contacts. 21.An article of manufacture comprising: a non-transitory computerprocessor readable medium; and instructions stored on the medium;wherein the instructions are configured to be readable from the mediumby at least one computer processor communicatively coupled to andconfigured to operate in a contact center system and thereby cause theat least one computer processor to operate so as to: during a firstperiod of time, associate a first pairing strategy to a first pluralityof contacts; during a second period of time later than the first periodof time, associate a second pairing strategy that is different from thefirst pairing strategy to a second plurality of contacts; during a thirdperiod of time later than the second period of time, associate a thirdpairing strategy that is different from the second pairing strategy to athird plurality of contacts; during a fourth period of time later thanthe third period of time, associate a fourth pairing strategy that isdifferent from the first and third pairing strategies to a fourthplurality of contacts; determine a first performance measurement basedon outcomes of the first and third pluralities of contacts; determine asecond performance measurement based on outcomes of the second andfourth pluralities of contacts; and output data that enables acomparison of the first and second performance measurements, wherein thefirst, second, third, and fourth periods of time occur within a timeinterval of less than 24 hours, wherein the first period of time isdetermined prior to the first period of time, wherein the second periodof time is determined prior to the second period of time, wherein thethird period of time is determined prior to the third period of time,wherein the fourth period of time is determined prior to the fourthperiod of time, wherein each of a plurality of agents is paired with atleast one contact from the first plurality of contacts, at least onecontact from the second plurality of contacts, at least one contact fromthe third plurality of contacts, and at least one contact from thefourth plurality of contacts.
 22. The article of manufacture of claim21, wherein the time interval is less than or equal to 12 hours.
 23. Thearticle of manufacture of claim 21, wherein the time interval is lessthan or equal to 6 hours.
 24. The article of manufacture of claim 21,wherein the time interval is less than or equal to 3 hours.
 25. Thearticle of manufacture of claim 21, wherein the time interval is lessthan or equal to 1 hour.
 26. The article of manufacture of claim 21,wherein the plurality of agents comprises at least 10 agents.
 27. Thearticle of manufacture of claim 21, wherein the plurality of agentscomprises at least 30 agents.
 28. The article of manufacture of claim21, wherein the plurality of agents comprises at least 100 agents. 29.The article of manufacture of claim 21, wherein the time interval isselected to reduce bias attributable to effects related to time.
 30. Thearticle of manufacture of claim 21, wherein a combination of the firstand third pluralities of contacts share a common distribution of contacttypes with a combination of the second and fourth pluralities ofcontacts.